OpenMed developed a pipeline that transforms protein concepts into codon-optimized DNA sequences. They compared various transformer architectures for codon-level language modeling, finding CodonRoBERTa-large-v2 to outperform others in biological relevance. The project includes detailed results and runnable code for each stage.

• CodonRoBERTa-large-v2 hit perplexity 4.10 and Spearman CAI correlation 0.40 across 381,000 sequences from 25 species, beating other tested architectures.
• ModernBERT badly underperformed classic RoBERTa on codon modeling (perplexity 26.24 vs 4.01), showing that advanced NLP architectures and pretrained weights don't transfer well to biological sequence data.
• The entire training pipeline ran on just 4 A100 GPUs for a total cost of $165.